Adaptive Semiparametric Language Models

Abstract We present a language model that combines large parametric neural network (i.e., transformer) with non-parametric episodic memory component in an integrated architecture. Our uses extended short-term context by caching local hidden states—similar to transformer-XL—and global long-term retrieving set of nearest neighbor tokens at each timestep. design gating function adaptively combine multiple information sources make prediction. This mechanism allows the use either context, memory, or (or any combination them) on ad hoc basis depending context. Experiments word-based and character-based modeling datasets demonstrate efficacy our proposed method compared strong baselines.